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FORWARD 


Any "comprehensive" system of planning and evaluation should always ensure a good balance of 
information sources and utilization which leads to effective measurement of the goals and objectives of the 
organization or program in question. That is, any information which is collected about the program should 
come from as many possible valid sources and should also be used or examined to the fullest extent, when 
resources make it feasible to do so. 


The establishment of such an information system must also ensure that the goals and objectives of the 
program/service have been clearly stated before evaluation activities can be meaningful. Using this notion as 
a conceptual backdrop, this paper will focus on two important development issues: 


1. Developing an information system which serves effectively in the measurement of the 
established goals and objectives of a program, service, and/or organization; 


and 


Zs Automating (through microcomputers) information systems for more effective and efficient 
planning and evaluative purposes. 


A "primer" is usually characterized as a small introductory book useful for teaching general or 
elementary concepts about a specific topic. In reading through this document, it is important to keep this 
definition in mind. It is not a how-to manual - it is a resource document, meant to give managers a conceptual 
"starting point" in the development and implementation of any information system used for planning in a non- 
profit organization. 


The responsibility to get "hands-on" and more knowledgeable about any evolving database system lies 
with the respective personnel who will be using it. More sophisticated or technically advanced issues cannot 
necessarily be addressed through this publication. However, the basic elements noted herein can help any 
manager to identify other resource or information sources that could be of more direct assistance. 
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1.0 DEVELOPING INFORMATION SYSTEMS 


Any system of information within an organization should be capable of reporting and reflecting upon 
its stated goals and objectives. 


"Goals" and "objectives" are defined as follows: 
GOALS What, generally, the program or service is designed to accomplish. 


OBJECTIVES: Established and detailed statements of process or outcome in which descriptive terms 
are specific, measurable, and time-based. 


A. Administrative and Service Objectives 


There are two general types of organizational/program objectives: 


1) Administrative Objectives 


= Mostly concerned with the development of "quality assurance” and "total quality management" 
practices. 

Resource allocation and acquisition. 

Communications within and outside the organization. 

Morale (staff, volunteer, etc.). 

Community awareness. 


* * & 


2) Service Objectives 


a) Implementation (Process) 


bs The level or quality of service being planned and expected. 
F The means by which outcomes will be achieved. 
Identifying the client groups that will be served. 


b) Outcome 
: What the "process" will achieve. 
. The impacts (positive or negative) of these processes/actions. 
. Client benefits and client numbers. 


B. Common Problems in Measuring Objectives 


Having identified different types of measurable objectives, some common problems in measuring these 
objectives often become evident. 


One problem, often described by evaluators, is that programs are defined using supposed "objective" 
statements which are really not clearly stated; in fact, they are more like generalized goal statements. The 
distinction between processes and outcomes (means and ends) is often non-existent and, frequently, outcomes 
are not even stated, but implied. Most important to the evaluator is the need for "specificity" in stating 
objectives. If they are not immediately measurable, they are probably not specific enough to be effectively 
evaluated. 

1. 


There are a few suggested guidelines planners can follow to make objectives more measurable: 


Always create objective statements using terms that are easily understood by all who will be responsible 
for their achievement. 

Clearly distinguish between processes and outcomes in creating service objectives. 

Focus on the main aspects of the program or service to be evaluated. These will become the critical 
components or "questions" you will derive for subsequent evaluation. 

Refer to a single process or outcome for each objective statement and identify "performance criteria" 
for each. That is, specifically, how each objective will be accomplished and to what degree. 

Establish the degree to which objectives are to be immediate, short-term, intermediate, or long-term 
in nature, with corresponding timelines or terms of reference for their achievement. 


C. Creating the Information System 


Having linked the importance of clearly defined goals and objectives to that of effective evaluation and 


measurement in non-profit organizations, the next question is "How do we systematically measure these same 
objectives?". Essentially, every measurement system must have information or “data” which can be utilized for 
decision-making. 


* 


* 


* 


An information system is best characterized by the following statement: 


It is a collection of information which is continually updated and utilized for planning or evaluative 
purposes. 


Any evaluative information system includes: 


Standardized input: all information entering the system must be standardized using a common 
methodology for collection, common or similar terminology, and standard recording procedures, 
regardless of whether it is quantitative or qualitative information. 


Continuous input: Information must be constantly updated and at regular intervals to be relevant (and 
reliable) for planning purposes. 


Continuous output: Summarized information must be analyzed and interpreted on a continual and 
regular basis in order to effectively monitor process and outcome (the immediate to longer term nature 
of each objective being measured will play a role in the intervals for evaluating the information 
collected). 


Information systems also comprise the following components: 
Input: Actual pieces of information, either quantitative or qualitative, automated or manually processed, 


which are collected ard utilized for evaluative purposes. 


Analysis: Making information useful and interpretable; that is, data are easy to read and understand, 
easily maintained in a standardized form, and immediately useful for analysis. 


. Output: Reporting the outcomes, analysis and interpretation of the data; being able to generate reports 
which demonstrate the outcomes, processes, or impacts which were measured. 


Information systems can also be classified in terms of using two types of data: 


* Resource Data: Data which describe human, material, or financial resources and are focused on the 
administrative aspects of the organization or service. 


- Client Data: Can also be termed program or service data; intended to measure processes and/or 
outcomes of service provision as they relate to client impact. 


D. Connecting Information Systems with Effective Evaluation and Planning 


Having created a standardized system of information, it then becomes important to utilize these data 
sources in addressing the evaluative question: 


"What kind of client benefits most (or least) from this program?" 


It is possible to illustrate the relevant types of data or information that might be collected to address 
each aspect (or objective) in this question, if one were to break it up into three separate parts (refer to Figure 


Ly 
What kind of client... 


Generally, clients can be described in terms of their own personal characteristics and also the situational 
or environmental condition(s) in which they find themselves. Information sources can range from basic 
demographic/socio-economic data about the client or client group to situational descriptors of, for example, 
quality of life conditions and pre-program expectations. More in-depth information can lead to the description 
of the client’s problem and its severity relative to other clients or standards which have been established. 


benefits most (or least)... 


Client (or program) impact measures determine the degree to which those targeted by the program or 
innovation have been effected by its activities. These measures may focus on either process or outcomes 
resulting from the change(s), or lack thereof, brought on by the activity. Measures may include client 
satisfaction, change in situational/environmental descriptors, change in problem severity, and so on. 


from this program? 


This includes the description of the program’s structure and how it functions to affect change as it was 
designed, as well as describing the extent to which service or results were achieved, as stated in the program’s 
objectives. 


Revisiting the evaluative question mentioned earlier and keeping Figure 1 in mind can thus yield relevant 


planning information for: 
dL 


and 


WHAT KIND OF 


CLIENT 


BENEFITS 
THE MOST 


FROM THIS 
PROGRAM 


Directing your resources to those areas or on those clients which/who will benefit the most; 


Developing new approaches for those clients who benefit the least from existing programs. 


- Demographics 

- Situational descriptors 
- Problem descriptors 

- Problem severity 


- Client satisfaction 
- Change in situational descriptors 
- Change in problem severity 


- Program descriptors 
- Amount of service received 


Figure 1:_ Information Requirements 


4. 


2.0 DEVELOPING COMPUTER SYSTEMS FOR EVALUATION 


There are a number of steps one can follow to bring a computerized system of information to 
operational (and functional) status. In order to determine how and what data will need to be entered and 
processed, it is important, as noted in the first section of this document, to recognize the outputs to be achieved 
in answering critical questions about your program or service. 


The advent of the microcomputer has made in-house computer systems much more accessible and 
realistic for even small applications. However, there is a down-side to this new technology. In the 1988 book 
Managing Your Information,Carol Tenopir and Gerald Lundeen note the following: 


Designers of in-house databases often want to immediately choose the hardware and software for their 
project and get quickly to the stage of inputting records. Those are certainly the steps that are the most 
rewarding in terms of seeing a real product, but they are prone to frustration, delay and sometimes even 
failure if the proper groundwork is not laid first. 1 


The process referred to above is applicable to any organization, regardless of the level of 
computerization. You may be doing everything manually and have no computers, you may have a computer 
which is only used for word processing, or, you may have several computers and have several systems running. 


If you are planning to introduce a new computer system,regardless of whether it is a brand new system 
to carry out a function that was previously done manually or a new system to replace a current computer system, 
this procedure should help you plan its implementation more realistically. 


A. Advantages to Developing a Computer System 


There are a number of advantages in working through such a process: 


dL. It should help in planning a timetable for implementation. The development of computer systems can 
take a lot of time, and knowing the steps required can help in completing specific tasks and can keep 
the evaluator better informed as to the progress of the new system. 


oF By doing the analyses first, before beginning to use a computer package, one ends up with a system that 
functions according to those needs. Buying a package first and then trying to "fit" your data into a non- 
specific system will only create frustration. 


3. Such a process will help in identifying the personnel required to further develop the system and what 
skills they will need. You may be able to do a portion of the work yourself and then discover a need 
for an outside resource to do the programming, for example. Or, you may require somebody to lead 
you through the first stages and, if you have somebody on staff who is happy doing the computer work, 
you can do the latter stages of data entry and analysis yourself. 


4. It will force you, the manager, to think more clearly about staffing; who will be the end users of the 
computer system, what will their tasks be and how will they be trained? 


> You will end up with a better understanding of what it is you are actually trying to do. You should also 
end up with some good documentation to explain your procedures, and provide continuity through staff 
changes. 


1 Carol Tenopir and Gerald Lundeen, Managing Your Information, Neal-Shuman Publishers, Inc., New 
York, 1988. 


B. Key Steps to Developing a Computer System 


The development of any computer system for use in planning and evaluation generally follows a logical 
sequence of phases or steps, including the following: Analysis, Design, Programming, Testing, Training, 
Conversion, Parallel Run, and Documentation. 


1. Analysis 


1) Inventory 


This is where information needs are determined; the information you want to keep, how you want to 
update it, what reports are required, and what analysis is required. When computerizing existing procedures, 
the best way to start is to do an inventory of the system. 


First, collect all the reports that are produced in the organization (note: this may be either a manual 
or automated system) and note from whence the data comes. Some information will need input directly, such 
as "name" and “address”, while other information will be calculated; such as, “how long has the client been 
involved in a programme”, which can be calculated using the current date and the date the client first arrived. 


Collect all the forms that are used: the ones first used to record information about the client. This may 
be in paper form, or you may want to enter it directly into a computer. Print out the screen formats, if 
necessary. Other forms may be filled in later to communicate information to someone else. Note all the 
procedures that take place; for instance, what happens when a client is first introduced to the program and where 
does the client go next? 


2) Requirements 


Having done the inventory, the next task is to determine your new requirements. Start with the output. 
What information do you need to get out of the system? Some of it may duplicate your existing reports, but you 
may identify new reports. 


Defining the output will help you determine what information you need to capture or to input into the 
system. Remember that not all information will be input into the system. Some of it will be calculated. 
Consider the selection techniques. Do you want all clients on all reports, or only some? How will each type 
be recognized? You may have to think about classification codes, and related "valid" (or meaningful) values. 


All of this leads to identifying the data elements that will be required in your system. A data element 
is simply an item of information, such as a surname, first name, date of birth, etcetera. Other terms used to 
describe data elements include "fields" or "variables". 


Consider the activities that will take place on a daily basis (such as updating client information), on a 
weekly basis, on a monthly basis, or on an annual basis within your program. For example, on a monthly basis, 
you may wish to produce a report showing all clients who have been on file for six months, and who need 
assessment, or, who may need to fill in a questionnaire regarding program satisfaction. How will you identify 
those clients? Probably by looking at their respective start dates. Or, you may classify clients and only wish to 
assess a certain type of client each month. Then you would set up a data element, perhaps titled "client type” 
and call up this variable by a certain value each month. 


During the analysis process, you have the opportunity to change your procedures. To this point, you 
may have been locked into a specific way of doing things. Now is your chance to add to it, or to decide whether 
a report is really necessary. After doing so, you should be very familiar with how your current system works, 
and what you want to do with the new system. Now you are ready to get into the specifics of system design. 


2. Design 
1) Output 


Once you know your output needs, it is time to lay them out in detail. Reports need titles, dates, page 
numbers, and headings. Some reports require breaks and/or sub-totals, (e.g., how many clients are within a 
particular age group). You need to decide in what order the reports will come out, (e.g., alphabetical, by 
surname, or client type order, or client age). This is called the sort order. 


You need to look at field size, (e.g., three digits is enough for age, but how many characters are required 
for surname), in order to determine the layout of the report. Will everything fit on the same line or do you have 
to do some manipulation? 


2) Database Design 


Having identified the data elements, the items to be input at their source and which fields are calculated 
need to be determined. You will need to define whether a data element is character or numeric. If it is 
numeric, what is the format? How many decimal places are required? 


The valid values or ranges for each data element must be defined. For example, the client type 
discussed earlier may have three or four values and any other values are unacceptable. The valid value may be 
in a range. A date, for example, can be checked for a range of 1 to 12 by months, 1 to 31 by days, etc. 


It is important to think about how to calculate fields or how fields are processed. Where do the fields 
that are used in the calculation come from? Once this is done, you can assign names of limited character length 
to each field. 


3) Processing Routines 


In the design stage, the processing routines are also written out. For example, if you plan to produce 
a report showing a specific type of client with a specific need as of a specific date (say, in the past six months), 
you will have to instruct the system to look for a client with a type equal to a value *p1GP4faandate equal to the 
current date less six months. 


Updating routines is another important consideration. How will clients be added to the system? What 
information can change and what information is "fixed"? How will clients be removed from the system and after 
how long? If all client files are updated automatically each month, what fields are updated? What statistics are 
kept as clients are added into the system? 


At the end of the design stage, a document which outlines the new system as a whole should be 
produced. It should list the data element names, data element descriptions, processing routines, edit rules, and 
the output. This information can then be transferred to the individual(s) responsible for setting up the system 
on the computer for data entry and manipulation (analysis) within the computer environment (usually, through 
use of a specific software program). 


3. Programming 


An important consideration in being able to effectively program and analyze any data set or system is 
the actual database management software that will be utilized. There are specific criteria that can be employed 
in the determination of which software package(s) would best meet the needs of your organization in managing 
and reporting on client/program impacts. 


Criteria for evaluating database management systems software should be based on the needs of the 
organization. One type of evaluation that may be used is the weighted-rating matrix method. In this type of 
evaluation, each criterion receives a weight according to its importance for the agency. In another Evaluation 
Consultation Program publication, entitled Evaluating Database Management Systems (Doyle, 1993), this process 
of rating is described in specific detail. Drawing upon this research paper, some suggested criteria can be 
established: 


e Price 

e Performance 

e Ease of Use 

e Help 

e Security 

e Professional Applications Development 
e Recovery Services 

© Scaleability 

e Import/Export Capabilities 

e Statistical Analysis 


Not all of these criteria may apply to every organization. 
Price is simply the cost of the package. 


Performance concerns the speed of data accessing and loading, the use of expanded and extended 
memory and the economy of disk space. It may also cover batch order processing and the speed of 
importing and exporting files. When evaluating performance, it is helpful to look at the series of 
benchmark tests carried out by computer magazines such as PC Magazine. 


Ease of Use covers the package’s quality and usability of end-user tools and the ability to get data in and 
out without programming. To score high under this criterion the package should offer a friendly 
Graphical User Interface (GUI). 


Help covers documentation, vendor assistance and end-user and on-line help functions. 


Security should be judged on the varying levels of password protection, encryption capabilities and the 
package’s ability to restrict unwanted "read" and "write" functions of parts of data. 


Professional applications development concerns the package’s availability of powerful tools for the 
programmer to develop the package to meet the needs of the user. 


Recovery services is the ability of the package to recover the database in the event that it is damaged 
in any way. 


Scaleability is based on the ability of a software package to grow with the organization. For example, 
can the package be upgraded to a Local Area Network (LAN) version for future expansion? 
Scaleability also includes the package’s support of multi-users and the availability of multi-user support 
functions. 


Import/Export capability pertains to the transfer of files to and from word processing, statistical 
analysis and other database management software packages. 


Statistical Analysis refers to whether the package allows for simple statistical functions, (e.g., frequency 
tables, crosstabulations, correlation) without programming. 


Once you have made an informed decision to select a software package, the database can be set up using 
the information which has been defined in the newly created systems design document. Input screens are set-up 
and reports defined through specific programming "syntax" or software language. Edit rules (or, how you want 
the data to be read and reported) are programmed, as well as any processing routines. The actual how-to will 
vary from package to package. Learning the in’s and out’s of the software can sometimes become time 
consuming - this is where use of built-in tutorials is most helpful before initiating any software applications. 


4. Testing 


Once programming has been completed, it is then necessary to test the program for "bugs" (also called 
cleaning) before depending on it as a trusted information source. This allows one to determine where any errors 
in programming and/or reporting have been committed by the programmer. 


A good approach to testing any computer program involves the creation of "dummy data" (data that are 
meaningless, describing hypothetical clients, but which have values falling within the programmed value ranges 
of the data elements in the system file). By generating mock cases or files and then combining data in as many 
different ways as possible, the analyst can then begin to see patterns emerge and/or errors which arise from 
programming inconsistencies. 


By entering incorrect or inaccurate information, the system’s "logic" is tested to ensure the integrity of 
any reports that will be generated later on. In other words, testing enables the programmer/analyst to have both 
valid and reliable measurement values for evaluation purposes. This will become especially critical whenever 
evaluation reaches a point of determining inferential statistical outcomes and/or causal relationships in terms 
of client impacts. 


Ultimately, testing computer programs is an iterative process. The programmer/analyst finds an error, 
corrects that error and goes on in a logical sequence of cleaning until an acceptable level of confidence is 
reached in processing the data. 


5._ Training 


It is unusual to find that all end-users of a database system were also involved in the development of 
the software programming or the detailed design stages of the database. Thus, training becomes an important 
and necessary component. Non-profit agencies are unique in that it may sometimes be community volunteers 
who are asked to enter or manage data. Training is an ongoing process, especially where there is a high turnover 
rate of volunteers (or staff). One alternative is to identify one person within the organization as the "trainer" 
who can immediately begin the task of training other individuals on the system. It is preferable that training take 
a "one-to-one" approach in order to see more direct and immediate results. Where this is not possible (e.g., not 
enough expertise, not enough human resources...), an alternative is to develop a manual which is "user friendly" 
and which allows the end-user the opportunity to make useful reference to it as required, without extra searching 
or reading (often, the standard manuals purchased with the software package are unrealistic and too technical). 
Creating your own custom made user manual can often be very instructive to the person designated as the 
"trainer". 


In some organizations, it may not be important that all end-users are trained to the same level of 
proficiency with either the database system or the software used to manage that system. For some users, being 
able to enter and clean data is all that is required. At the opposite end of the user spectrum, there may be a 
planner who needs to know more advanced functions and applications of the software and who has the 
responsibility to ask constructive/objective questions of the data being collected. Again, it is important that all 
levels of users have back-up persons familiar with that level of complexity, should their input be required. 


9. 


10. 


A final point under training is that a back-up or test system should always be maintained specifically for 
the purpose of training new individuals in the system. Simply having a file or two of dummy data, with the same 
data elements and programming schema provides a new user with a simulated environment for practicing 
software applications safely and without risk of lost data sets. 


6. Conversion 


Once the new system has been tested, it is then necessary to determine how much information from the 
existing system has to be input into the new system. Usually, the data which will be entered relies upon some 
set and measurable objectives which have been asked by the evaluator or planner. The actual amount of 
information which is entered and processed within the system is dependent on the complexity of these objectives 
and the degree to which the resulting output will answer (in part or whole) some "critical questions" about the 
data. Thus, the amount of information which is entered does not necessarily equal the amount of information 
which has actually been collected. 


Additionally, it is important to determine the amount of time that can be afforded in carrying out data 
entry, cleaning, and manipulation. 


7. Parallel Run 


When an existing database is being transcribed from one format or environment to a new software 
system, e.g., manual to automated or one software package to another, it is important to ensure the integrity of 
the new software system by carrying out a parallel runexercise. That is, both systems are run and checked with 
each other simultaneously to determine where differences may lie in formatting, processing and so on. While 
this may seem to be a time-consuming and arduous task, it can prove itself an effective technique in ensuring 
that error does not invade your new system (or perpetuate itself!) in the transition. 


Data processing errors may arise, despite all the prior testing of the new system. It is much safer to 
have a "default" system to fall back upon than to risk compromising the integrity of previously collected and 
programmed data sets. The duration of a parallel run will vary depending on the type of system being developed. 
For some systems one week will suffice, while others may require at least one month. It is advisable to work 
through the normal cycle of information collection, entry, and processing for the program in question. 


The key is to measure the personal confidence levels of end users and their comfort with the new 
system. When it seems that users are starting to question the necessity of carrying both systems (new and old) 
it is likely time to close out the old system. 


8. Documentation 


One area which is often neglected in the process of database management is documentation of the 
system, especially in organizational environments where there is likely to be high turnover of users. Generally, 
after the initial design stages have been completed and a basic user manual has been described, it is important 
to be able to document all aspects of the system’s creation and ongoing development. Describe the "nuts and 
bolts" of the system, the diagnostic considerations which are important in keeping the system maintained and 
functioning fully, and the reports and procedures to be carried out on a regular basis. These are essential pieces 
of information to obtain and update as required. 


3.0 CONCLUSION 


Developing a comprehensive and useful information system takes a serious commitment of time, energy 
and thought. It is important to remember that this document alone cannot possibly provide you, the 
planner/programmer with all the insight necessary to develop and maintain a database management system. 
However, it should at least provide any program manager with a starting point to understanding the basic 
elements and approaches to building such a system. 


What becomes apparent is the need to become practical and hands-on in one’s view of developing such 
a system. That is, if the system is not utilized actively and by those who would benefit most from its existence, 
it can become cumbersome and ineffective. The old saying, "start small, but think big” is a useful adage for 
managers to consider regarding any system development activities. The best systems are built in steps or phases, 
the context of which depends on the resource capabilities of the organization. It is unusual to find any two 
organizations that would have the same information systems functioning at identical levels of output or 


productivity. 


The benefits of following the above-mentioned guidelines will be demonstrated time after time. 
Ultimately, the confidence in decision-making which can be achieved from the development of an in-house 
database which is well-managed and reliable will validate the need for all program managers and/or planners 
to incorporate such systems wherever and whenever viable. 
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